Large-scale data storage has shifted from a central-service architecture to distributed storage systems. Distributed storage systems built from commodity computers can deliver high performance, availability, and scalability at a fraction of the cost compared to monolithic disk arrays. Data is replicated across multiple instances of the distributed storage system at different geographical locations, thereby increasing availability and reducing network distance from clients.
In a distributed storage system, objects are dynamically placed in (i.e., created in, deleted from, and/or moved to) various instances of the distributed storage system based on constraints. There are few existing techniques for efficiently placing objects that are subject to constraints in a planet-wide distributed storage system that stores trillions of objects and petabytes of data, and includes dozens of data centers across the planet.
New visualization, multimedia, and other data-intensive applications use very large objects, which may be hundreds of gigabytes or larger. Managing such very large objects create additional complexities for a distributed storage system. First, uploading such an object into a distributed storage system is typically done in a streaming mode, splitting the object into chunks and writing each chunk individually. This can impose long delays for uploading, which is exacerbated by potential client and server failures. Moreover, chunks may be aggregated into larger shards for better operational efficiency. The terms “shard” and “journal” may be used interchangeably herein. Consequently, efficient upload of large objects is becoming increasingly important for the storage industry being driven by the needs of large-scale systems that allow clients to connect to any cluster available at a time. In addition, the volume of metadata for a single object (e.g., 25000 chunks for a 100 gigabyte file, where each chunk is 4 megabytes) makes replication and compaction less efficient.